This pull request addresses the issue of inconsistent batch size calculation during multi-GPU training. Previously, the number of batches per epoch did not correctly account for the number of GPUs, leading to an incorrect batch size. The fix involves adjusting the data parallel world size and rank when the model parallel unit (mpu) is not defined. This ensures that the number of batches is correctly calculated as the training data size divided by the number of GPUs, aligning with the expected behavior. The changes are made in the deepspeed/runtime/engine.py file.

Created by Genie. You can follow its reasoning on Cosine